Computer technology has continued to advance at a remarkable pace, with each subsequent generation of a computer system increasing in performance, functionality and storage capacity, and often at a reduced cost. A modern computer system typically comprises one or more central processing units (CPU) and supporting hardware necessary to store, retrieve and transfer information, such as communication buses and memory. A modern computer system also typically includes hardware necessary to communicate with the outside world, such as input/output controllers or storage controllers, and devices attached thereto such as keyboards, monitors, tape drives, disk drives, communication lines coupled to a network, etc.
From the standpoint of the computer's hardware, most systems operate in fundamentally the same manner. Processors are capable of performing a limited set of very simple operations, such as arithmetic, logical comparisons, and movement of data from one location to another. But each operation is performed very quickly. Sophisticated software at multiple levels directs a computer to perform massive numbers of these simple operations, enabling the computer to perform complex tasks. What is perceived by the user as a new or improved capability of a computer system is made possible by performing essentially the same set of very simple operations, but doing it much faster, and thereby enabling the use of software having enhanced function. Therefore continuing improvements to computer systems require that these systems be made ever faster.
The overall speed of a computer system (also called the throughput) may be crudely measured as the number of operations performed per unit of time. Conceptually, the simplest of all possible improvements to system speed is to increase the clock speeds of the various components, and particularly the clock speed of the processor(s). E.g., if everything runs twice as fast but otherwise works in exactly the same manner, the system will perform a given task in half the time. Enormous improvements in clock speed have been made possible by reduction in component size and integrated circuitry, to the point where an entire processor, and in some cases multiple processors along with auxiliary structures such as cache memories, can be implemented on a single integrated circuit chip. Despite these improvements in speed, the demand for ever faster computer systems has continued, a demand which can not be met solely by further reduction in component size and consequent increases in clock speed. Attention has therefore been directed to other approaches for further improvements in throughput of the computer system.
Without changing the clock speed, it is possible to improve system throughput by using a parallel computer system incorporating multiple processors that operate in parallel with one another. The modest cost of individual processors packaged on integrated circuit chips has made this approach practical. Although the use of multiple processors creates additional complexity by introducing numerous architectural issues involving data coherency, conflicts for scarce resources, and so forth, it does provide the extra processing power needed to increase system throughput, given that individual processors can perform different tasks concurrently with one another.
Various types of multi-processor systems exist, but one such type of system is a massively parallel nodal system for computationally intensive applications. Such a system typically contains a large number of processing nodes, each node having its own processor or processors and local (nodal) memory, where the nodes are arranged in a regular matrix, or lattice structure. The system contains a mechanism for communicating data among different nodes, a control mechanism for controlling the operation of the nodes, and an I/O mechanism for loading data into the nodes from one or more I/O devices and receiving output from the nodes to the I/O device(s). In general, each node acts as an independent computer system in that the addressable memory used by the processor is contained entirely within the processor's local node, and the processor has no capability to directly reference data addresses in other nodes. However, the control mechanism and I/O mechanism are shared by all the nodes.
A massively parallel nodal system such as described above is a general-purpose computer system in the sense that it is capable of executing general-purpose applications, but it is designed for optimum efficiency when executing computationally intensive applications, i.e., applications in which the proportion of computational processing relative to I/O processing is high. In such an application environment, each processing node can independently perform its own computationally intensive processing with minimal interference from the other nodes. In order to support computationally intensive processing applications which are processed by multiple nodes in cooperation, some form of inter-nodal data communication matrix is provided. This data communication matrix supports selective data communication paths in a manner likely to be useful for processing large processing applications in parallel, without providing a direct connection between any two arbitrary nodes. Optimally, I/O workload is relatively small, because the limited I/O resources would otherwise become a bottleneck to performance.
An exemplary massively parallel nodal system is the IBM Blue Gene®/L (BG/L) system. The BG/L system contains many (e.g., in the thousands) processing nodes, each having multiple processors and a common local (nodal) memory, and with five specialized networks interconnecting the nodes for different purposes. The processing nodes are arranged in a logical three-dimensional torus network having point-to-point data communication links between each node and its immediate neighbors in the network. Additionally, each node can be configured to operate either as a single node or multiple virtual nodes (one for each processor within the node), thus providing a fourth dimension of the logical network. A large processing application typically creates one or more blocks of nodes, herein referred to as communicator sets, for performing specific sub-tasks during execution. The application may have an arbitrary number of such communicator sets, which may be created or dissolved at multiple points during application execution. The nodes of a communicator set typically comprise a rectangular parallelopiped of the three-dimensional torus network.
The hardware architecture supported by the BG/L system and other massively parallel computer systems provides a tremendous amount of potential computing power, e.g., petaflop or higher performance. Furthermore, the architectures of such systems are typically scalable for future increases in performance. However, unless the software applications running on the hardware architecture operate efficiently, the overall performance of such systems can suffer.
As an example, BG/L system performance can be hindered by various conditions. Communication bottlenecks between nodes can result from poor network utilization or ported code. Other problems may be attributable to incorrect assumptions about communication nodal matrix geometries. For instance, a designated path between nodes may be longer than it should be, resulting in a load imbalance or link contention. Poor performance may likewise result from cache misses and/or temperature-related problems.
It is consequently incumbent upon system designers and administrators to locate and fix such problems. When assessing performance in massively parallel computers, administrators conventionally gather and evaluate performance data in one of two ways. Using the first technique, administrators simultaneously run one or more performance tools on all of the nodes of the system. The performance data is aggregated and later evaluated. Although this conventional technique provides comprehensive results, the simultaneous burden from analysis on all of the nodes can result in a tremendous overhead on the entire system.
Administrators analyzing performance data using the second technique will conventionally run a performance tool(s) on a single node. Information gleaned from the tool is used to infer performance with regard to the remaining nodes of the lattice. This technique, however, disproportionally burdens the sample node. The resultant overhead on the node can result in slowed system performance, particularly where the monitored node is disproportionately burdened and impacts the overall amount of work that can be done.
To counteract the overhead generated by these conventional techniques, analysts often compromise performance assessments by reducing the sampling rate. A conventional sampling tool may periodically initiate (“wake up”), e.g., every one-five milliseconds, and determine what instruction the processor each node is executing. The handling of the initiating and recording of the sample data may result in overhead that can detrimentally affect system performance. As a consequence, performance analysts reduce the sample rate to lessen the impact on the system being tested. This compromise, however, causes the analyst to have to wait longer to get their results. Since performance analysis is often an iterative process, any additional waiting for each collection can result in significant dissatisfaction for the analyst and a larger bill for a client.
Therefore, a need exists for an improved manner of assessing performance of a plurality of interconnected nodes of a parallel computer system.